Papers with Fine-tuning pre-trained models
Fisher Mask Nodes for Language Model Merging (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained models are ubiquitous in natural language processing, but individual fine-tuned models require significant overhead in multi-task scenarios. |
| Approach: | They propose a method for fine-tuning pre-trained models for Transformers using Fisher information. |
| Outcome: | The proposed method outperforms Fisher-weighted averaging in a fraction of the computational cost. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |